Skip to content

Add publication monitoring and revise the frontier panel#75

Merged
nedcut merged 20 commits into
mainfrom
codex/publication-run-monitor
Jul 18, 2026
Merged

Add publication monitoring and revise the frontier panel#75
nedcut merged 20 commits into
mainfrom
codex/publication-run-monitor

Conversation

@nedcut

@nedcut nedcut commented Jul 16, 2026

Copy link
Copy Markdown
Owner

What changed

  • add a read-only publication-run monitor with one-shot, watch, and JSON modes
  • replace the stale ten-model registry with the user-curated twelve-model frontier panel
  • pin every OpenRouter cell by exact provider slug, endpoint tag, and dated endpoint name
  • support per-model reasoning provenance: disabled where optional and lowest supported effort where mandatory
  • provision a common 4,096-token smoke ceiling with a predeclared 3,072-token pressure trigger and 8,192 fallback
  • reset the superseded 1,024-token smoke manifest and preserve its run ledger as historical evidence
  • regenerate the conservative cost plan and public leaderboard gate

Revised panel

Proprietary: GPT-5.6 Luna, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Muse Spark 1.1.

Open-weight: GLM 5.2, Kimi K3, Nemotron 3 Ultra, MiniMax M3, Qwen 3.7 Plus, DeepSeek V4 Pro, Mistral Medium 3.5.

Terra, Sol, Claude Opus 4.8, and Fable 5 are recorded as future candidates.

Routing and reasoning

  • provider fallbacks disabled
  • required-parameter filtering and JSON mode enabled
  • data-collection routes denied
  • exact endpoint tags validated before model execution
  • reasoning disabled for eight optional-reasoning models
  • Gemini and Muse use minimal; Grok uses low; Kimi K3 uses max, its only catalog-supported effort
  • Claude uses healthy amazon-bedrock/global because the direct Anthropic route was unhealthy during selection
  • Nemotron Ultra uses Together rather than the cheaper DeepInfra FP4 route

All twelve no-spend live endpoint/credential preflights passed.

Cost plan

  • 5,760 full-panel model calls
  • conservative full-cap estimate: $194.63
  • 1.2x contingency: $235.50
  • twelve smoke reservations at full-cap assumptions: $1.62

Refresh the estimate from accepted smoke telemetry before approving the full panel.

Validation

  • 443 passed
  • Ruff format and lint clean
  • benchmark validity canaries pass
  • web lint and production build pass
  • all model execution remains explicitly serial with workers=1

Summary by CodeRabbit

  • New Features

    • Added publication-run status reporting with progress, cost/reservation tracking, and per-model runtime details.
    • Added the ability to reuse previously validated smoke-test results when they still meet current criteria.
  • Updates

    • Updated the leaderboard publishing lane to use a 4,096-token safety ceiling with native-minimum reasoning (reasoning handling varies by model requirement).
    • Refreshed the model panel rules, pricing/cost estimates, and leaderboard “frozen” metadata.
  • Documentation

    • Updated publication-readiness guidance and added detailed smoke-run and panel revision records.

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@nedcut, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 52 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6697a9f4-9f31-485a-ac42-4f0f2ae3528e

📥 Commits

Reviewing files that changed from the base of the PR and between 57bf425 and 71fbe02.

📒 Files selected for processing (2)
  • gm_bench/publication.py
  • tests/test_publication_runner.py
📝 Walkthrough

Walkthrough

The PR replaces the provisional 1,024-token reasoning-disabled lane with a frozen 4,096-token native-reasoning panel, adds exact route and reasoning validation, introduces publication-run status and smoke-artifact reuse, and updates evidence, documentation, tests, and leaderboard metadata.

Changes

SOTA v2 publication lane

Layer / File(s) Summary
Frozen lane and smoke contracts
config/sota_v2_*.json, config/publication_protocol.json, config/openrouter_pricing_snapshot.json, docs/output_budget_sweep.md
The registry and lane define frozen routes, endpoint metadata, per-model reasoning policies, a 4,096-token cap, cap-pressure escalation, exclusions, pricing, and smoke requirements.
Exact route and reasoning validation
gm_bench/*.py, scripts/analyze_publication_panel.py, web/scripts/build_leaderboard.py
Validation now compares upstream slugs, endpoint tags, expected providers, fixed/absent options, and policy-specific reasoning telemetry.
Publication runner and status flow
scripts/run_publication_matrix.py, tests/test_publication_runner.py
The runner validates exact routes, records state, supports status/watch output, reuses valid smoke artifacts, settles reservations, and applies registered route settings.
Evidence and publication state
docs/**, results/analysis/output-budget-cost-estimate.json, web/src/data/leaderboard.json, tests/**
Cost estimates, readiness records, smoke ledgers, fixtures, publication tests, and leaderboard metadata reflect the revised lane and smoke-gate state.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related issues

Possibly related PRs

  • nedcut/gm-bench#52 — Shares OpenRouter route provenance and mismatch validation changes.
  • nedcut/gm-bench#61 — Shares publication runner, route validation, gating, and status-flow changes.
  • nedcut/gm-bench#70 — Shares smoke-manifest recording and validation changes.

Sequence Diagram(s)

sequenceDiagram
  participant Registry
  participant PublicationRunner
  participant SmokeManifest
  participant Leaderboard
  Registry->>PublicationRunner: supplies frozen routes and reasoning policies
  PublicationRunner->>SmokeManifest: records accepted route and telemetry metadata
  SmokeManifest->>Leaderboard: validates publication identity and smoke state
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 13.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the two main changes: publication monitoring and the revised frontier panel.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/publication-run-monitor

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@nedcut nedcut changed the title Add live publication-run monitoring and smoke evidence Add publication monitoring and revise the frontier panel Jul 16, 2026
@nedcut
nedcut marked this pull request as ready for review July 18, 2026 16:26
Copilot AI review requested due to automatic review settings July 18, 2026 16:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 10

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@config/sota_v2_models.json`:
- Around line 161-178: Remove the expiring route entry identified by id
openrouter-tencent-hy3-free-novita, or replace it with a stable, smoke-tested
route that remains valid beyond the freeze. Do not retain this route unless the
publication process is explicitly bounded before its catalog_expiration_date.
- Around line 14-15: The frozen model panel in the selection_policy and models
registry must contain twelve models: five proprietary and seven open-weight.
Update the model entries and associated required_smokes coverage so the declared
cohort, smoke completeness, cost planning, and leaderboard eligibility
consistently use the required five-plus-seven composition.

In `@docs/PUBLISH_READINESS.md`:
- Around line 13-17: Synchronize the frozen ten-model phase-one state across
docs/PUBLISH_READINESS.md lines 13-17, 30-32, 136-165, and 410, marking fresh
4,096-token smokes, verification, cost-plan, and lane-freeze work as complete
and removing provisional or retired-plan wording. Update
docs/output_budget_sweep.md lines 3-7 to call the current policy frozen, and
lines 62-65 to remove excluded Kimi K3 from the frozen lane’s reasoning
configuration.

In `@results/analysis/output-budget-cost-estimate.json`:
- Around line 127-136: Regenerate the runtime evidence in the output-budget cost
estimate from the accepted 4,096-token smoke observations, replacing the legacy
1,024-token source and updating the observation timestamp. Populate
observed_api_seconds_per_decision_by_model for all ten models in the accepted
ledger, rerun the estimator, and update the runtime status from
pending-smoke-telemetry to the resulting final evidence status.

In `@scripts/run_publication_matrix.py`:
- Around line 535-540: Keep the smoke-row state logic in
scripts/run_publication_matrix.py lines 535-540 based on current-run,
current-cap raw/checkpoint artifacts rather than manifest acceptance; expose
manifest acceptance separately or validate its artifact path/hash and cap before
using it. In tests/test_publication_runner.py lines 642-646, key the reservation
with cell.cap_label and assert the row-level reservation. In
tests/test_publication_runner.py lines 673-676, write raw artifacts under the
current 4,096-token stem so the monitor reads them.
- Around line 787-790: Update the reasoning-token validation around the
reasoning_policy checks to require reasoning_tokens to be a non-negative
integer, explicitly excluding booleans. Apply this validation to the
mandatory-minimum telemetry check and ensure negative values are rejected
wherever reasoning telemetry is validated, while preserving the existing
disabled-policy handling.
- Around line 995-999: Update the smoke reuse branch in the publication matrix
flow to skip reuse when args.preflight_only is enabled, ensuring that
_validate_openrouter_endpoint still runs for explicit live endpoint preflight
while preserving normal reuse behavior otherwise.
- Around line 87-100: Update the reasoning-policy validation around policy and
effort checks: require mandatory-minimum models to provide a non-null
reasoning_effort and matching OPENROUTER_REASONING_EFFORT, and reject disabled
models when fixed_options contains any reasoning-effort setting in addition to
enforcing disabled reasoning. Preserve the existing unknown-policy validation.

In `@web/scripts/build_leaderboard.py`:
- Around line 116-125: Update the upstream provider handling in the
expected_options construction to access registered["upstream_provider_slug"]
safely via the existing optional-value pattern, while preserving the current
OPENROUTER_PROVIDER_ONLY validation behavior when the slug is present.

In `@web/src/data/leaderboard.json`:
- Around line 83-86: Regenerate the leaderboard gate data from the frozen lane
and current model registry: update frozen_output_token_cap to reflect the
4,096-token policy and replace the obsolete embedded three-model list with the
current registry entries. Preserve the empty missing list and
model_registry_frozen state, and apply the same regenerated values to the
additional leaderboard records noted in the comment.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d144199d-9889-4b95-a46f-05ca95c7c6d0

📥 Commits

Reviewing files that changed from the base of the PR and between 6bc8523 and f7d4f58.

📒 Files selected for processing (27)
  • README.md
  • config/openrouter_pricing_snapshot.json
  • config/output_budget_sweep.json
  • config/publication_protocol.json
  • config/sota_v2_lane.json
  • config/sota_v2_models.json
  • config/sota_v2_smoke_manifest.json
  • docs/PUBLISH_READINESS.md
  • docs/output_budget_sweep.md
  • docs/run_logs/sota-v2-panel-revision-2026-07-16.md
  • docs/run_logs/sota-v2-smokes-2026-07-16.md
  • docs/run_logs/sota-v2-smokes-2026-07-17.md
  • gm_bench/cli.py
  • gm_bench/official.py
  • gm_bench/providers.py
  • gm_bench/publication.py
  • results/analysis/output-budget-cost-estimate.json
  • scripts/analyze_publication_panel.py
  • scripts/estimate_publication_cost.py
  • scripts/run_publication_matrix.py
  • tests/test_publication.py
  • tests/test_publication_analysis.py
  • tests/test_publication_cost.py
  • tests/test_publication_runner.py
  • tests/test_smoke_manifest.py
  • web/scripts/build_leaderboard.py
  • web/src/data/leaderboard.json
💤 Files with no reviewable changes (1)
  • tests/test_publication_cost.py

Comment on lines +14 to 15
"selection_policy": "User-curated current frontier panel: five large American-lab proprietary models and five leading open-weight models. Every OpenRouter route pins an exact provider slug, endpoint tag, and dated endpoint name with fallbacks disabled. Models run at native-minimum reasoning: disabled when optional and the lowest catalog-supported effort when mandatory. Kimi K3 is parked after its clean smoke failed and demonstrated cap pressure; the exact Nemotron 3 Ultra and DeepSeek V4 Pro routes are parked after bounded HTTP 404 infrastructure failures.",
"models": [

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

The frozen panel contains ten models instead of the declared twelve.

The registry and required_smokes define five proprietary plus five open-weight models, while the publication objective requires five plus seven. Freezing this state propagates the wrong cohort into smoke completeness, cost planning, and leaderboard eligibility.

Also applies to: 181-199

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@config/sota_v2_models.json` around lines 14 - 15, The frozen model panel in
the selection_policy and models registry must contain twelve models: five
proprietary and seven open-weight. Update the model entries and associated
required_smokes coverage so the declared cohort, smoke completeness, cost
planning, and leaderboard eligibility consistently use the required
five-plus-seven composition.

Comment on lines +161 to +178
"id": "openrouter-tencent-hy3-free-novita",
"provider": "openrouter",
"model": "tencent/hy3:free",
"transport": "gateway-api",
"cohort": "open-weight",
"upstream_provider": "Novita",
"upstream_provider_slug": "novita",
"endpoint_tag": "novita",
"endpoint_name": "Novita | tencent/hy3-20260706:free",
"catalog_expiration_date": "2026-07-21",
"reasoning_policy": "disabled",
"reasoning_effort": null,
"fixed_options": {
"OPENROUTER_REASONING_ENABLED": "false",
"OPENROUTER_JSON_MODE": "false"
},
"absent_options": ["OPENROUTER_REASONING_EFFORT"],
"role": "Tencent open-weight frontier anchor on OpenRouter's temporary free Novita route; prompt-only JSON because the endpoint does not advertise response_format"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Do not freeze a route that expires on July 21, 2026.

This route expires only four days after the July 17 freeze. Publication execution or reproduction after that date will fail unless the run is explicitly bounded before expiry or a stable replacement is selected and smoked.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@config/sota_v2_models.json` around lines 161 - 178, Remove the expiring route
entry identified by id openrouter-tencent-hy3-free-novita, or replace it with a
stable, smoke-tested route that remains valid beyond the freeze. Do not retain
this route unless the publication process is explicitly bounded before its
catalog_expiration_date.

Comment thread docs/PUBLISH_READINESS.md Outdated
Comment thread results/analysis/output-budget-cost-estimate.json Outdated
Comment thread scripts/run_publication_matrix.py
Comment thread scripts/run_publication_matrix.py Outdated
Comment thread scripts/run_publication_matrix.py
Comment thread scripts/run_publication_matrix.py Outdated
Comment thread web/scripts/build_leaderboard.py
Comment thread web/src/data/leaderboard.json Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/run_publication_matrix.py`:
- Around line 803-806: The mandatory-reasoning validation in
gm_bench.publication.smoke_manifest_issues must reject reasoning_tokens when it
is not an integer, is a boolean, or is negative, matching _record_smoke_issues
before allowing panel unlock. Update scripts/run_publication_matrix.py lines
803-806 accordingly; add coverage in tests/test_publication_runner.py lines
570-599 through the persisted-manifest or build_cells("panel") path for True and
-1.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 05900922-07e3-4c75-8a85-3b87b2202250

📥 Commits

Reviewing files that changed from the base of the PR and between f7d4f58 and 57bf425.

📒 Files selected for processing (10)
  • config/openrouter_pricing_snapshot.json
  • docs/PUBLISH_READINESS.md
  • results/analysis/output-budget-cost-estimate.json
  • scripts/estimate_publication_cost.py
  • scripts/run_publication_matrix.py
  • tests/test_publication.py
  • tests/test_publication_cost.py
  • tests/test_publication_runner.py
  • web/scripts/build_leaderboard.py
  • web/src/data/leaderboard.json
🚧 Files skipped from review as they are similar to previous changes (3)
  • results/analysis/output-budget-cost-estimate.json
  • config/openrouter_pricing_snapshot.json
  • docs/PUBLISH_READINESS.md

Comment on lines +803 to +806
if entry.get("reasoning_policy") == "mandatory-minimum" and (
not isinstance(reasoning_tokens, int) or isinstance(reasoning_tokens, bool) or reasoning_tokens < 0
):
issues.append("artifact is missing reasoning-token telemetry for a mandatory-reasoning model")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Apply the reasoning-token invariant at the panel-unlock gate.

_record_smoke_issues now rejects boolean and negative values, but smoke_manifest_issues still accepts them for mandatory-reasoning models. An edited invalid manifest can therefore unlock the panel.

  • scripts/run_publication_matrix.py#L803-L806: apply this same predicate in gm_bench.publication.smoke_manifest_issues.
  • tests/test_publication_runner.py#L570-L599: exercise the persisted-manifest or build_cells("panel") path with True and -1.
📍 Affects 2 files
  • scripts/run_publication_matrix.py#L803-L806 (this comment)
  • tests/test_publication_runner.py#L570-L599
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/run_publication_matrix.py` around lines 803 - 806, The
mandatory-reasoning validation in gm_bench.publication.smoke_manifest_issues
must reject reasoning_tokens when it is not an integer, is a boolean, or is
negative, matching _record_smoke_issues before allowing panel unlock. Update
scripts/run_publication_matrix.py lines 803-806 accordingly; add coverage in
tests/test_publication_runner.py lines 570-599 through the persisted-manifest or
build_cells("panel") path for True and -1.

@nedcut
nedcut merged commit cec75f8 into main Jul 18, 2026
13 checks passed
@nedcut
nedcut deleted the codex/publication-run-monitor branch July 19, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants